Design Of Validation Experiments Research Articles

Stress Knowledge Map: A knowledge graph resource for systems biology analysis of plant stress responses

Stress Knowledge Map (SKM, https://skm.nib.si) is a publicly available resource containing two complementary knowledge graphs describing current knowledge of biochemical, signalling, and regulatory molecular interactions in plants: a highly curated model of plant stress signalling (PSS, 543 reactions) and a large comprehensive knowledge network (CKN, 488,390 interactions). Both were constructed by domain experts through systematic curation of diverse literature and database resources. SKM provides a single entrypoint for plant stress response investigations and the related growth trade-offs as well as interactive exploration of current knowledge. PSS is also formulated as qualitative and quantitative models for systems biology, and thus represents a starting point of a plant digital twin. Here, we describe the features of SKM and show, through two case studies, how it can be used for complex analyses, including systematic hypothesis generation, design of validation experiments, or to gain new insights into experimental observations in plant biology.

Optimal design of validation experiments for the prediction of quantities of interest

Numerical predictions of quantities of interest measured within physical systems rely on the use of mathematical models that should be validated, or at best, not invalidated. Model validation usually involves the comparison of experimental data (outputs from the system of interest) and model predictions, both obtained at a specific validation scenario. The design of this validation experiment should be directly relevant to the objective of the model, that of predicting a quantity of interest at a prediction scenario. In this paper, we address two specific issues arising when designing validation experiments. The first issue consists in determining an appropriate validation scenario in cases where the prediction scenario cannot be carried out in a controlled environment. The second issue concerns the selection of observations when the quantity of interest cannot be readily observed. The proposed methodology involves the computation of influence matrices that characterize the response surface of given model functionals. Minimization of the distance between influence matrices allows one to select a validation experiment most representative of the prediction scenario. We illustrate our approach on two numerical examples. The first example considers the validation of a simple model based on an ordinary differential equation governing an object in free fall to put in evidence the importance of the choice of the validation experiment. The second numerical experiment focuses on the transport of a pollutant and demonstrates the impact that the choice of the quantity of interest has on the validation experiment to be performed.

Toward selecting optimal predictive multiscale models

This work presents a systematic strategy for selecting an “optimal” predictive computational model among a set of possible mechanistic models of a physical system. To this end, the Occam-Plausibility Algorithm (Farrell-Maupin et al., 2015) is extended by introducing a method for the design of model-specific validation experiments to provide data that portray the features of the prediction quantities of interest. Leveraging Bayesian inference and model plausibility, the framework adaptively balances the trade-off between complexity and validity of the models while taking into account the uncertainty in data, model parameters, and the choice of the model itself for making reliable computational predictions. An application of this framework is demonstrated in discrete-continuum multiscale modeling of size-dependent plasticity in polycrystalline materials, involving dislocation dynamics simulations and strain gradient plasticity models.This study suggests that the effectiveness of validation experiments relies upon the choice of the model; thus, a validation data set that adequately informs a predictive computational model may not be effective for validating other models. Additionally, the multiscale modeling results show that, given the discrete dislocation dynamics data, the optimal strain gradient plasticity model for predicting the deformation of a microelectromechanical system is obtained by excluding the isotropic hardening mechanism.

Optimization of Process Parameters for Biodiesel Production from Waste Cooking Oil using DOE Techniques

In the current study the process parameters for used cooking oil biodiesel production are optimized by using Design of Experiments (Taguchi technique) and validation experiments are carried out at the optimized parameters to cross verify the results obtained. A specially designed equipment which can maintain the exact conditions required for optimization is used. This equipment is microcontroller operated and does not require any human intervention to maintain the accurate temperature, time and other parameters. By Taguchi strategy best yield acquired is by Molar ratio of A2 (1:9), Catalyst concentration of B3 (1.00%), Reaction temperature of C2 (60ºC), Reaction time of D3 (120 min). The approval trial was completed for the results and the yield is observed to be 93.25%, which resulted in improvement of up to 15% yield as compared to that of crude method. The properties of acquired biodiesel are found out and it very well may be inferred that the properties of got biodiesel are within ASTM norms.

A Response-Adaptive Method for Design of Validation Experiments in Computational Mechanics

For model verification and validation (V & V) in computational mechanics, a hypothesis test for the validity check (HTVC) is useful, in particular, with a limited number of experimental data. However, HTVC does not address how type I and II errors can be reduced when additional resources for sampling become available. For the validation of computational models of safety-related and mission-critical systems, it is challenging to design experiments so that type II error is reduced while maintaining type I error at an acceptable level. To address the challenge, this paper proposes a new method to design validation experiments, response-adaptive experiment design (RAED). The RAED method adaptively selects the next experimental condition from among candidates of various operating conditions (experimental settings). RAED consists of six key steps: (1) define experimental conditions, (2) obtain experimental data, (3) calculate u-values, (4) compute the area metric, (5) select the next experimental condition, and (6) obtain additional experimental datum. To demonstrate the effectiveness of the RAED method, a case study of a numerical example is shown. It is demonstrated that additional experimental data obtained through the RAED method can reduce type II error in hypothesis testing and increase the probability of rejecting an invalid computational model.

Multi-Resolution Sensitivity Analysis of Model of Immune Response to Helicobacter pylori Infection via Spatio-Temporal Metamodeling

Computational immunology studies the interactions between the components of the immune system that includes the interplay between regulatory and inflammatory elements. It provides a solid framework that aids the conversion of pre-clinical and clinical data into mathematical equations to enable modeling and in silico experimentation. The modeling-driven insights shed lights on some of the most pressing immunological questions and aid the design of fruitful validation experiments. A typical system of equations, mapping the interaction among various immunological entities and a pathogen, consists of a high-dimensional input parameter space that could drive the stochastic system outputs in unpredictable directions. In this paper, we perform spatio-temporal metamodel-based sensitivity analysis of immune response to Helicobacter pylori infection using the computational model developed by the ENteric Immune SImulator. We propose a two-stage procedure to obtain the estimates of the Sobol’ total and first-order indices for each input parameter, for quantifying their time-varying impacts on each output of interest. In particular, we fully reuse and exploit information from an existing simulated dataset, develop a novel sampling design for constructing the two-stage metamodels, and perform metamodel-based sensitivity analysis. The proposed procedure is scalable, easily interpretable, and adaptable to any multi-input multi-output complex systems of equations with a high-dimensional input parameter space.

Optimal Design of Validation Experiments Based on Area Metric Factor and Fuzzy Expert System

Experiments used for model validation constitute a new type of experiments, whose primary goal is to determine the ability of a high-fidelity mathematical model in simulating a well-characterized physical process. Considering the importance of the credibility of the validation experiment, in this paper, a methodology for the optimal design of validation experiments based on the area metric factor and the fuzzy expert system is presented to obtain the test scheme with low cost and satisfactory credibility. First, the concept of area metric factor is proposed, which is a dimensionless validation metric. Then, a criterion for qualitative assessment of the predictive model accuracy is developed based on the fuzzy expert system, in which the inconsistency among the different expert groups is considered. Subsequently, an optimization model of the validation experiment design is constructed, in which the sample size of the experiments is designed variable, while the credibility of the validation experiment is a constraint. Meanwhile, a novel method is developed to solve the optimization model, in which the Latin hypercube sampling is used to obtain the location of each validation experiment. Finally, two simulation examples are adopted to validate the proposed methods. The simulation results show that the randomness of the experimental scheme significantly affects the credibility of the evaluation results in the case of small sample sizes. The proposed method for the validation experiment design can reduce the impact of the selection of the testing programs on the credibility of model validation.

Capillary microsampling in nonclinical safety assessment: practical sampling and bioanalysis from a CRO perspective.

Microsampling has the 3R benefits of refining blood collection techniques while reducing the number of animals required for rodent safety assessment studies. There are significant scientific benefits of correlating study findings with systemic exposure and consequently, there is an industry drive to utilize microsampling in regulatory toxicology studies. This article will focus on capillary microsampling and will discuss the initial considerations before using capillary microsampling, study design and some practicalities of sample collection. From a bioanalytical perspective, the impact of different sample collection and storage workflows on the design of method validation experiments, the suitability of using microsampling for unstable analytes and aspects supporting analysis of peptides, large molecules and for biomarker applications will be discussed.

Design of validation experiments for life prediction models

This paper proposes a novel validation experiment design optimization (VEDO) method for the assurance of life prediction model, which is one of the key steps in guaranteeing the reliable design of products in meeting the target service life. Life testing data collected from experiments are important for the validation of time-dependent models. However, directly collecting life data for model validation at the operating stress level is usually time-consuming and expensive. In order to overcome this challenge, the accelerated life testing (ALT) method is employed in the proposed method to collect data for model validation. The connection between ALT and model validation is established first; then a VEDO model is developed using the prior information obtained from the computer simulation model. In the VEDO model, the information gain for model validation is maximized within the testing budget and available testing chamber constraints. The obtained optimal number of tests and testing stress levels are designed to maximize the confidence in the validation results. Various sources of uncertainty such as prediction uncertainty, uncertainty of prior information, and observation errors are included within the optimization process in order to improve the robustness of validation experiment design. A composite helicopter rotor hub component is used to demonstrate the effectiveness of the proposed VEDO method.

Selection Criterion Based on an Exploration-Exploitation Approach for Optimal Design of Experiments

AbstractModeling and simulation are being relied upon in many fields of science and engineering as computational surrogates for experimental testing. To justify the use of these simulations for decision making, however, it is critical to determine, and when necessary mitigate, the biases and uncertainties in model predictions, a task that invariably requires validation experiments. To use experimental resources efficiently, validation experiments must be designed to achieve the maximum possible increases in model predictive ability through the calibration of the model against experiments. This need for efficiency is addressed by the concept of optimally designing validation experiments, which constitutes optimizing a predefined criterion while selecting the settings of experiments. This paper presents an improved optimization criterion that incorporates two important factors for the optimal design of validation experiments: (1) how well the model reproduces the validation experiments, and (2) how well the...

Multi-omic landscape of rheumatoid arthritis: re-evaluation of drug adverse effects.

Objective: To provide a frame to estimate the systemic impact (side/adverse events) of (novel) therapeutic targets by taking into consideration drugs potential on the numerous districts involved in rheumatoid arthritis (RA) from the inflammatory and immune response to the gut-intestinal (GI) microbiome.Methods: We curated the collection of molecules from high-throughput screens of diverse (multi-omic) biochemical origin, experimentally associated to RA. Starting from such collection we generated RA-related protein-protein interaction (PPI) networks (interactomes) based on experimental PPI data. Pharmacological treatment simulation, topological and functional analyses were further run to gain insight into the proteins most affected by therapy and by multi-omic modeling.Results: Simulation on the administration of MTX results in the activation of expected (apoptosis) and adverse (nitrogenous metabolism alteration) effects. Growth factor receptor-bound protein 2 (GRB2) and Interleukin-1 Receptor Associated Kinase-4 (IRAK4, already an RA target) emerge as relevant nodes. The former controls the activation of inflammatory, proliferative and degenerative pathways in host and pathogens. The latter controls immune alterations and blocks innate response to pathogens.Conclusions: This multi-omic map properly recollects in a single analytical picture known, yet complex, information like the adverse/side effects of MTX, and provides a reliable platform for in silico hypothesis testing or recommendation on novel therapies. These results can support the development of RA translational research in the design of validation experiments and clinical trials, as such we identify GRB2 as a robust potential new target for RA for its ability to control both synovial degeneracy and dysbiosis, and, conversely, warn on the usage of IRAK4-inhibitors recently promoted, as this involves potential adverse effects in the form of impaired innate response to pathogens.

Statistical methods and software for validation studies on new in vitro toxicity assays.

When a new in vitro assay method is introduced, it should be validated against the best available knowledge or a reference standard assay. For assays resulting in a simple binary outcome, the data can be displayed as a 2×2 table. Based on the estimated sensitivity and specificity, and the assumed prevalence of true positives in the population of interest, the positive and negative predictive values of the new assay can be calculated. We briefly discuss the experimental design of validation experiments and previously published methods for computing confidence intervals for predictive values. The application of the methods is illustrated for two toxicological examples, by using tools available in the free software, namely, R: confidence intervals for predictive values are computed for a validation study of an in vitro test battery, and sample size calculation is illustrated for an acute toxicity assay. The R code necessary to reproduce the results is given.

Non-lethal control of the cariogenic potential of an agent-based model for dental plaque.

Dental caries or tooth decay is a prevalent global disease whose causative agent is the oral biofilm known as plaque. According to the ecological plaque hypothesis, this biofilm becomes pathogenic when external challenges drive it towards a state with a high proportion of acid-producing bacteria. Determining which factors control biofilm composition is therefore desirable when developing novel clinical treatments to combat caries, but is also challenging due to the system complexity and the existence of multiple bacterial species performing similar functions. Here we employ agent-based mathematical modelling to simulate a biofilm consisting of two competing, distinct types of bacterial populations, each parameterised by their nutrient uptake and aciduricity, periodically subjected to an acid challenge resulting from the metabolism of dietary carbohydrates. It was found that one population was progressively eliminated from the system to give either a benign or a pathogenic biofilm, with a tipping point between these two fates depending on a multiplicity of factors relating to microbial physiology and biofilm geometry. Parameter sensitivity was quantified by individually varying the model parameters against putative experimental measures, suggesting non-lethal interventions that can favourably modulate biofilm composition. We discuss how the same parameter sensitivity data can be used to guide the design of validation experiments, and argue for the benefits of in silico modelling in providing an additional predictive capability upstream from in vitro experiments.

Defining coverage of an operational domain using a modified nearest-neighbor metric

Validation experiments are conducted at discrete settings within an operational domain to assess the predictive maturity of a model that is ultimately used to predict over the entire operational domain. Unless this domain is sufficiently explored with validation experiments, satisfactory model performance at these discrete, tested settings would be insufficient to ensure satisfactory model performance throughout the entire operational domain. The goal of coverage metrics is then to reveal how well a set of validation experiments represents an operational domain. The authors identify the criteria of an exemplary coverage metric, evaluate the ability of existing coverage metrics to fulfill these criteria, and propose a new, improved coverage metric. The proposed metric favors interpolation over extrapolation through a penalty function, causing the metric to prefer a design of validation experiments near the boundaries of the domain, while simultaneously exploring inside the domain. Furthermore, the proposed metric allows the coverage to account for the relative influence of each dimension of the domain on the model output. Application of the proposed coverage metric on a practical, non-trivial two-dimensional problem is demonstrated on the Viscoplastic Self-Consistent material plasticity code for 5182 aluminum alloy. Furthermore, the proposed metric is compared to existing coverage metrics considering a high dimensional problem with application to the Rosenbrock function.

MiRvestigator: web application to identify miRNAs responsible for co-regulated gene expression patterns discovered through transcriptome profiling

Transcriptome profiling studies have produced staggering numbers of gene co-expression signatures for a variety of biological systems. A significant fraction of these signatures will be partially or fully explained by miRNA-mediated targeted transcript degradation. miRvestigator takes as input lists of co-expressed genes from Caenorhabditis elegans, Drosophila melanogaster, G. gallus, Homo sapiens, Mus musculus or Rattus norvegicus and identifies the specific miRNAs that are likely to bind to 3′ un-translated region (UTR) sequences to mediate the observed co-regulation. The novelty of our approach is the miRvestigator hidden Markov model (HMM) algorithm which systematically computes a similarity P-value for each unique miRNA seed sequence from the miRNA database miRBase to an overrepresented sequence motif identified within the 3′-UTR of the query genes. We have made this miRNA discovery tool accessible to the community by integrating our HMM algorithm with a proven algorithm for de novo discovery of miRNA seed sequences and wrapping these algorithms into a user-friendly interface. Additionally, the miRvestigator web server also produces a list of putative miRNA binding sites within 3′-UTRs of the query transcripts to facilitate the design of validation experiments. The miRvestigator is freely available at http://mirvestigator.systemsbiology.net.

Spatiotemporal dynamics of the Calvin cycle: Multistationarity and symmetry breaking instabilities

AbstractThe possibility of controlling the Calvin cycle has paramount implications for increasing the production of biomass. Multistationarity, as a dynamical feature of systems, is the first obvious candidate whose control could find biotechnological applications. Here we set out to resolve the debate on the multistationarity of the Calvin cycle. Unlike the existing simulation-based studies, our approach is based on a sound mathematical framework, chemical reaction network theory and algebraic geometry, which results in provable results for the investigated model of the Calvin cycle in which we embed a hierarchy of realistic kinetic laws. Our theoretical findings demonstrate that there is a possibility for multistationarity resulting from two sources, homogeneous and inhomogeneous instabilities, which partially settle the debate on multistability of the Calvin cycle. In addition, our tractable analytical treatment of the bifurcation parameters can be employed in the design of validation experiments.

Well-Posed Model Uncertainty Estimation by Design of Validation Experiments

In deterministic model validation approaches, model errors can be attributed to both disturbances and model uncertainty, leading to an ill-posed problem formulation. The aim of this paper is to remedy the ill-posedness in model validation for robust control. A two-stage procedure is developed, where first an accurate, nonparametric, deterministic disturbance model is estimated from data, followed by the enforcement of averaging properties through an appropriate periodic experiment design. The proposed deterministic approach results in an asymptotically correctly estimated model uncertainty and is illustrated in a simulation example.

Bayesian cross-entropy methodology for optimal design of validation experiments

An important concern in the design of validation experiments is how to incorporate the mathematical model in the design in order to allow conclusive comparisons of model prediction with experimental output in model assessment. The classical experimental design methods are more suitable for phenomena discovery and may result in a subjective, expensive, time-consuming and ineffective design that may adversely impact these comparisons. In this paper, an integrated Bayesian cross-entropy methodology is proposed to perform the optimal design of validation experiments incorporating the computational model. The expected cross entropy, an information-theoretic distance between the distributions of model prediction and experimental observation, is defined as a utility function to measure the similarity of two distributions. A simulated annealing algorithm is used to find optimal values of input variables through minimizing or maximizing the expected cross entropy. The measured data after testing with the optimum input values are used to update the distribution of the experimental output using Bayes theorem. The procedure is repeated to adaptively design the required number of experiments for model assessment, each time ensuring that the experiment provides effective comparison for validation. The methodology is illustrated for the optimal design of validation experiments for a three-leg bolted joint structure and a composite helicopter rotor hub component.

Measuring CAMD technique performance: A virtual screening case study in the design of validation experiments

The dynamic nature and comparatively young age of computational chemistry is such that novel algorithms continue to be developed at a rapid pace. Such efforts are often wrought at the expense of extensive experimental validations of said techniques, preventing a deeper understanding of their potential utility and limitations. Here we address this issue for ligand-based virtual screening descriptors through design of validation experiments that better reflect the aims of real world application. Applying the newly defined chemotype enrichment approach, a variety of two- and three-dimensional (2D/3D) similarity descriptors have been compared extensively across data sets from four diverse target types. The inhibitors within said data sets contain molecules exhibiting a wide array of substructure functionality, size and flexibility, permitting descriptor comparison in myriad settings. Relative descriptor performance under these conditions is examined, including results obtained using more typical virtual screening validation experiments. Guidelines for optimal application of said descriptors are also discussed in the context of the results obtained, as is the potential utility of fingerprint filtering.

Modelling of the photooxidation of toluene: conceptual ideas for validating detailed mechanisms

Abstract. Toluene photooxidation is chosen as an example to examine how simulations of smog-chamber experiments can be used to unravel shortcomings in detailed mechanisms and to provide information on complex reaction systems that will be crucial for the design of future validation experiments. The mechanism used in this study is extracted from the Master Chemical Mechanism Version 3 (MCM v3) and has been updated with new modules for cresol and g-dicarbonyl chemistry. Model simulations are carried out for a toluene-NOx experiment undertaken at the European Photoreactor (EUPHORE). The comparison of the simulation with the experimental data reveals two fundamental shortcomings in the mechanism: OH production is too low by about 80%, and the ozone concentration at the end of the experiment is over-predicted by 55%. The radical budget was analysed to identify the key intermediates governing the radical transformation in the toluene system. Ring-opening products, particularly conjugated g-dicarbonyls, were identified as dominant radical sources in the early stages of the experiment. The analysis of the time evolution of radical production points to a missing OH source that peaks when the system reaches highest reactivity. First generation products are also of major importance for the ozone production in the system. The analysis of the radical budget suggests two options to explain the concurrent under-prediction of OH and over-prediction of ozone in the model: 1) missing oxidation processes that produce or regenerate OH without or with little NO to NO2 conversion or 2) NO3 chemistry that sequesters reactive nitrogen oxides into stable nitrogen compounds and at the same time produces peroxy radicals. Sensitivity analysis was employed to identify significant contributors to ozone production and it is shown how this technique, in combination with ozone isopleth plots, can be used for the design of validation experiments.